Time-of-flight camera system and method to improve measurement quality of weak field-of-view signal regions

ABSTRACT

A time-of-flight camera system is described. The time-of-flight camera system includes an illuminator to illuminate a region within the time-of-flight camera system&#39;s field of view. The time-of-flight camera system includes an image sensor to receive optical signals from the illumination for determining depth profile information within the field of view using time-of-flight measurement techniques. The image sensor has circuitry to determine one or more regions within the field of view where a received optical signal from the illuminating was weak. The illuminator is also to re-illuminate the one or more regions with stronger light than the one or more regions received during the illuminating. Each of the one or more regions being smaller than the region. The image sensor is also to receive optical signals from the re-illumination for determining depth profile information within the one or more regions.

FIELD OF INVENTION

The field of invention pertains to camera systems generally, and, morespecifically, to a time-of-flight camera system and method to improvemeasurement quality of weak field-of-view signal regions.

BACKGROUND

Many existing computing systems include one or more traditional imagecapturing cameras as an integrated peripheral device. A current trend isto enhance computing system imaging capability by integrating depthcapturing into its imaging components. Depth capturing may be used, forexample, to perform various intelligent object recognition functionssuch as facial recognition (e.g., for secure system un-lock) or handgesture recognition (e.g., for touchless user interface functions).

One depth information capturing approach, referred to as“time-of-flight” imaging, emits light from a system onto an object andmeasures, for each of multiple pixels of an image sensor, the timebetween the emission of the light and the reception of its reflectedimage upon the sensor. The image produced by the time of flight pixelscorresponds to a three-dimensional profile of the object ascharacterized by a unique depth measurement (z) at each of the different(x,y) pixel locations.

As many computing systems with imaging capability are mobile in nature(e.g., laptop computers, tablet computers, smartphones, etc.), theintegration of a light source (“illuminator”) into the system to achievetime-of-flight operation presents a number of design challenges such ascost challenges, packaging challenges and/or power consumptionchallenges.

SUMMARY

A time-of-flight camera system is described. The time-of-flight camerasystem includes an illuminator to illuminate a region within thetime-of-flight camera system's field of view. The time-of-flight camerasystem includes an image sensor to receive optical signals from theillumination for determining depth profile information within the fieldof view using time-of-flight measurement techniques. The image sensorhas circuitry to determine one or more regions within the field of viewwhere a received optical signal from the illuminating was weak. Theilluminator is also to re-illuminate the one or more regions withstronger light than the one or more regions received during theilluminating. Each of the one or more regions being smaller than theregion. The image sensor is also to receive optical signals from there-illumination for determining depth profile information within the oneor more regions.

An apparatus is described having means for illuminating a region withina time-of-flight camera system's field of view. The apparatus includesfirst means for determining depth profile information within the fieldof view using time-of-flight measurement techniques. The apparatusincludes second means for determining one or more regions within thefield of view where a received signal from the illuminating was weak.The first means includes means for re-illuminating the one or moreregions with stronger light than the one or more regions received duringthe illuminating. Each of the one or more regions is smaller than theregion. The second means includes means for determining depth profileinformation within the one or more regions.

FIGURES

The following description and accompanying drawings are used toillustrate embodiments of the invention. In the drawings:

FIGS. 1a (i) and 1 a(ii) pertain to a first possible smart illuminationfeature;

FIGS. 1b (i), 1 b(ii) and 1 b(iii) pertain to a partitioned smartillumination approach;

FIGS. 1c (i) and 1 c(ii) also pertain to a partitioned smartillumination approach;

FIGS. 1d (i) and 1 d(ii) pertain to another possible smart illuminationfeature;

FIG. 1e shows an embodiment of a light source for a partitioned field ofview;

FIGS. 2a through 2e pertain to scanning within a smart illuminationsystem;

FIG. 3 shows an embodiment of a smart illumination system;

FIGS. 4a through 4c show an embodiment of a smart illumination method;

FIG. 5 shows a first illuminator embodiment;

FIG. 6 shows a second illuminator embodiment;

FIG. 7 shows a third illuminator embodiment;

FIGS. 8a and 8b show a fourth illuminator embodiment;

FIG. 9 shows a 2D/3D camera system;

FIG. 10 shows a computing system.

DETAILED DESCRIPTION

A “smart illumination” time-of-flight system addresses some of thedesign challenges referred to in the Background section. As will bediscussed below, smart illumination involves the intelligentmanipulation of any or all of the size, shape or movement of the emittedlight of a time-of-flight system. Time-of-flight systems, and inparticular time-of-flight systems that are integrated in a batterypowered system, generally exhibit a tradeoff between the demands made onthe power supply and the emitted and received strength of the opticalsignals.

That is, as the illuminated optical signal strength grows, the receivedoptical signal strength improves. Better received optical signalstrength results in better accuracy and performance of thetime-of-flight system. However, supporting higher emitted optical signalstrength results in a more expensive battery solution and/or greaterdrain on battery life, either of which can be a drawback on userenjoyment and/or acceptance of a system having time-of-flightmeasurement capability.

Smart illumination strives to address this issue by concentratingilluminated optical power into smaller areas of illumination that aredirected on regions of interest in the camera field of view. Byconcentrating the optical power into smaller areas of illumination,received optical signal strength and time-of-flight system performanceis enhanced without increasing the power draw from the battery. As such,the aforementioned user perceived drawbacks may be acceptably minimized.

Use of smaller regions of light involves the ability to direct smallerregions of illumination to regions of interest within the camera's fieldof view. A region of interest is, for example, an area within the fieldof view of the camera that is smaller than the camera's field of viewand that is higher priority, in terms of obtaining depth information,than other areas within the field of view. Examples of a region ofinterest include a region where an object exists whose depth informationis desired (e.g., a hand, a face) or a region where a previously takentime-of-flight measurement yielded poor received signal intensity.

As such, after a region of interest within the field of view isidentified, the illumination system receives information indicative ofthe region of interest and concentrates optical intensity upon theregion of interest. Concentrating optical intensity upon the regioninterest may involve emitting optical light at or near a power limit ofthe illuminator but directing that light primarily upon the region ofinterest.

A first example includes, for an illuminator having a single lightsource, emitting light from the light source at a power limit of theilluminator and focusing a smaller “spot” of the light at the region ofinterest. A second example includes, for an illuminator having multiplelight sources, emitting light from one of the light sources at a powerlimit of the illuminator such that other ones of the light sources mustbe kept off and directing a beam of light from the illuminated lightsource toward the region of interest.

Other smart illumination schemes may take advantage of the illuminationof a smaller region of interest by illuminating the region of interestwith less than full illuminator power. For example, if a region ofinterest is small enough, sufficiently accurate information about theregion may be obtainable with less than full illuminator power.

Possible features of various smart illumination systems are discussed atlength below. In general, however, smart illumination systems may bedesigned to change either or both of the size and shape of anilluminated region in order to illuminate an object of interest withinthe camera's field of view. Additionally, smart illumination systems maybe designed to change the location of an illuminated region, e.g., byscanning an emitted beam within the field of view.

FIGS. 1a through 1d as discussed below pertain to aspects of changingilluminated region size and shape. By contrast, FIGS. 2a through 2c asdiscussed below pertain to aspects of changing the location of anilluminated region.

FIGS. 1a (i) and 1 a(ii) demonstrate that the size of an illuminatedregion may be adjusted in view of an object of interest to beilluminated. That is, in a first scenario 111 of FIG. 1a (i), a first,smaller object of interest 102 consumes a smaller area within thecamera's field of view 101. As such, the size of the illuminated regionof interest 103 that is emitted by the illuminator is contracted toencompass the smaller object of interest 102. By contrast, in scenario112 of FIG. 1a (ii), a second, larger object of interest 104 consumes alarger area within the camera's field of view 101. As such, the size ofthe illuminated region of interest 105 that is emitted by theilluminator is expanded to encompass the larger object of interest 104.

The contraction and expansion of the size of the illuminated region 103,105 can be accomplished, for example, with an illuminator having amovable optical component (e.g., a movable light source, a movable lens,a movable mirror, etc.). The controlled movement of an optical componentwithin the illuminator can be used to controllably set the size of theilluminated region. Examples of illuminators having movable opticalcomponents are discussed in more detail further below with respect toFIGS. 5a through 5 c.

Alternatively, as observed in FIGS. 1b (i) and 1 b(ii), the field ofview 101 can be partitioned into different sections that can beindividually illuminated (e.g., as observed in FIG. 1b (i), the field ofview is partitioned into nine different sections 106_1 through 106_9).As observed in scenario 121 of FIG. 1b (i), a smaller object of interest107 is illuminated by illuminating one of the partitions (partition106_1). By contrast, as observed in scenario 122 of FIG. 1b (ii), alarger object of interest 109 is illuminated by illuminating four of thepartitions (partitions 106_1, 106_2, 106_4 and 106_5). As such, theilluminated region of interest 110 of FIG. 1b (ii) is noticeably largerthan the illuminated region of interest 108 of FIG. 1b (i).

Referring to FIG. 1b (ii) note that the entire field of view can beilluminated by illuminating all partitions simultaneously, or, byilluminating each partition individually in succession (or some mixtureof the two approaches). The former approach is apt to induce weakerreceived optical signals. The later approach may be performed withhigher optical concentrations on the individually illuminated partitionsbut at the expense of the time needed to “scan” the field of view. Here,illuminating each partition in succession corresponds to a form ofscanning. Scanning is described in more detail further below withrespect to FIGS. 2a through 2 e.

The illumination of partitioned regions within the field of view can beaccomplished with a partitioned light source. FIG. 1b (iii) depicts atop view (looking into the face of the illuminator) of an exemplarypartitioned illuminator light source 117 having nine individual lightsources 113_1 through 113_9. In various embodiments, each individuallight source is implemented as an array of vertical cavity side emittinglasers (VCSELs) or light emitting diodes (LEDs) and is responsible forilluminating a particular partition. All of the individual light sources113_1 through 113_9 may be integrated, for example, on a samesemiconductor chip. In an embodiment, each individual light source isimplemented as an array of light source devices (VCSELs or LEDs) so thatthe entire illuminator power budget can be expended illuminating only asingle partition (in which case, the individual light sources for theother partitions must be off).

If illuminator light source 117 is used for scenario 121 of FIG. 1b (i),individual light source 113_1 would be on in order to illuminatepartition 106_1. By contrast, if illuminator light source 117 is usedfor scenario 112 of FIG. 1b (i), individual light sources 113_1, 113_2,113_4 and 113_5 would be on. More details about illuminators having apartitioned field of view and corresponding light source embodiments aredescribed in more detail further below.

FIG. 1c (i) shows another partitioned approach in which the partitionsthemselves are not all of the same size. That is, there exist differentpartitions having different sizes. With partitions of different sizes,the size of the illuminated region can be changed by illuminatingdifferent sized partitions in sequence. For example, if only a smallerpartition is illuminated and then only a larger partition is illuminatedthe size of the illuminated region will expand. An exemplary lightsource 119 for the partitioning approach of FIG. 1c (i) is observed inFIG. 1c (ii). Note that individual light sources that illuminate largerpartitions have greater potential optical power output (e.g., asdemonstrated by having more VCSELs or LEDs) than individual lightsources that illuminate smaller partitions.

Note that the expansion and/or contraction of the size of theilluminated region for a same emitted optical power (whether bynon-partitioned or partitioned mechanisms) involves a trade-off betweenthe size of the illuminated region and the strength of the receivedsignal. That is, for a same emitted optical power, a smaller illuminatedregion corresponds to a stronger received signal intensity. By contrast,again for a constant emitted optical power, a larger illuminated regioncorresponds to a weaker received signal.

If a larger illuminated region size is desired but without the loss inreceived signal strength, another tradeoff exists between theilluminated region size and the amount of power that will be consumed bythe illuminator. That is, in order to increase the size of theilluminated region but maintain the received optical signal strength,the illuminator (without any scanning as described below) will generallyneed to emit more intense light which will cause the illuminator toconsume more power.

Some smart illumination systems may be designed to maintain a minimumreceived optical signal strength at the image sensor. In situationswhere the illuminated region of interest contracts the emitted opticalintensity may be reduced because a sufficiently strong optical intensityper unit of illuminated surface area can still be maintained. Converselyilluminator power may increase with an expansion of the size of theregion of interest.

Additionally, in situations where the region of interest is smallerbecause the object to be illuminated is farther away, the emittedoptical intensity may reduce only slightly, remain constant or evenincrease because received optical signal is generally inverselyproportional with distance of the reflecting object from the camera. Assuch, a smart illumination system, besides considering the size of anobject of interest may also consider its distance when determiningappropriate illuminator optical power. A more thorough discussion of thefactors that a smart illumination system may consider when setting theillumination characteristics are described in more detail below withrespect to FIG. 3.

In various embodiments the shape of the illuminated region can bechanged. FIG. 1d (i) shows a first scenario 131 is which a pointed beamwhen directed in the middle of the field of view is essentiallycircular, but, as observed in scenario 132 of FIG. 1d (ii), when thesame beam is pointed to the corner of the field of view the illuminatedregion becomes more elliptical in shape. Movement of a pointed beam canbe accomplished with an illuminator having a movable optical componentas will be discussed in more detail below.

FIG. 1e shows an embodiment of a light source for a partitioned field ofview approach where the partitions themselves have different shapes. Assuch, illuminating only a first partition having a first shape, a thenonly illuminating only a second partition having a second, differentshape will correspondingly produce illuminated regions within the fieldof view that change shape.

FIGS. 1a through 1e discussed above pertain to smart illuminationsystems that can change the size and/or shape of the illuminated regionas the system attempts to suitably illuminate the different objects ofinterest that may be present in the field of view of a time-of-flightcamera system.

By contrast, FIGS. 2a through 2c pertain to the scanning of emittedlight within the camera's field of view. Scanning involves theintelligent changing of regions that receive illumination over time inorder to capture an overall region of interest that is larger than thesize(s) of the illuminated region(s) themselves. Here, recall from thediscussion of FIGS. 1a through 1e above that as the region of interestexpands, emitted optical intensity may have to be increased in order tomaintain sufficiently strong illumination and corresponding receivedsignal strength. Conceivably some regions of interest may be largeenough where the appropriate emitted optical intensity exceeds a desiredor permitted power budget for the illuminator.

Scanning helps maintain or enhance received signal strength over largerregions of interest but without corresponding increases in emittedoptical power intensity. That is, for example, by scanning anilluminated smaller “spot” over a larger region of interest, depthinformation can be collected for the larger region of interest eventhough optical power that is sufficient only to illuminate the smallerspot is expended.

FIG. 2a shows an example of scanning as described just above. Asobserved in FIG. 2a , a larger object of interest 205 is illuminated byscanning a smaller illuminated region initially from position 206 attime T1 to position 207 at time T2 and then to position 208 at time T3and finally to position 209 at time T4. The scanning of FIG. 2a may beachieved, for example, with an illuminator having a moving opticalcomponent that is able to point and sweep a beam of light in a scanningmotion within the field of view.

Alternatively or in combination, as observed in FIG. 2b , scanning maybe achieved with a partitioned illumination system by illuminatingdifferent partitions in an on and off sequence. That is, as observed inFIG. 2b , a first partition 211 is illuminated at time T1. Subsequently,at time t2, the first partition 211 is turned off and a second partition212 is turned on. Similar sequences subsequently occur at times t3 andt4 for third and fourth partitions 213, 214. Accordingly, a region ofinterest within all four partitions can be illuminated over times t1through t4.

FIG. 2c shows that scanning may be disjointed. That is, the embodimentsof FIGS. 2a and 2b assumed that a next region to be illuminated in ascan was adjacent to a previous region that was just illuminated. Bycontrast, FIG. 2c illustrates that a scan may include illuminating twoseparate regions that are not adjacent. Here, at time T1 a first region221 is illuminated. Then, at time T2, a second region 222 is illuminatedwhere the two regions are not adjacent to one another within the fieldof view. Disjointed scanning may be performed, for example, when a“region of interest” includes two or more different, non adjacent areasor items within the field of view that need to be illuminated. Disjointscanning may be performed with partitioned as well as non partitionedillumination schemes.

Note that the example of FIG. 2c also shows that the size of theilluminated region may change over a scanning sequence (illuminatedregion 222 is larger than illuminated region 221). Changing illuminatedregion size over the course of a scan is not limited to disjointed scansand may be a feature of continuous scans such as the scans of FIGS. 2aand 2b discussed above. In the case of partitioned scans, changing thesize of the illuminated region is possible, e.g., by first turning onone partition and then turning on multiple partitions.

FIG. 2d further illustrates that some partitioned smart illuminationsystems may be designed to perform scanning within a partition. That is,the illuminator may have both a partitioned light source and movableoptical components so that a smaller beam within a partition is scannedwithin the surface area of the partition to effectively illuminate thepartition. As observed in FIG. 2d , an illumination “spot” that issmaller than the size of a partition is scanned within the upper leftpartition to effectively illuminate the upper left partition. The entirefield of view can be scanned by scanning each partition in succession(sequentially) or simultaneously (as discussed further below withrespect to FIG. 2e ) or some mixture of these two approaches.

As discussed above, various illuminator embodiments are capable ofchanging the size of the illuminated region (by changing the crosssection of the bean of emitted light) while other illuminatorembodiments embrace a partitioned approach in which the field of view ispartitioned and the illuminated is capable of individually illuminatingeach partition. The approach of FIG. 2d can be integrated into anilluminator having both of these characteristics. That is, anilluminator whose design supports changing the size of the illuminatedregion can conceivably form a beam large enough to illuminate an entirepartition and also form a beam smaller than the entire partition so thatit can be scanned within the partition.

FIG. 2e shows another partitioned scanning approach in which respectivepartitions are scanned simultaneously with their own respective lightbeams. In an embodiment, the illuminator is designed to not only directa separate light beam to each partition simultaneously but also be ableto scan the light beams. Embodiments of an illuminator design capable ofsimultaneously scanning multiple partitions within the field of view aredescribed in more detail further below.

Note that although the embodiment of FIG. 2d is directed to apartitioned approach, other embodiments may scan over a region where theilluminator design does not fully embrace a partitioned designed (e.g.,a particular beam of light can be directed anywhere in a field of view).Simultaneous scanning of multiple beams, however, includes each beamhaving its own respective region in which to scan. Such regions may bedeemed to be partitions during the simultaneous multiple beam scanningsequence.

Any of the scanning approaches of FIGS. 2a through 2e discussed abovemay introduce a tradeoff between the time it takes to collect thetime-of-flight information for a region of interest and the size of theregion of interest. That is, for a constant illuminated region size(e.g., a “spot size”), as the size of the region of interest to beilluminated grows more scanning time will be consumed. Contra-wise, ifthe region of interest is increased, scanning time can be reduced byincreasing the size of the illumination but at the expense of increasedemitted optical power (if optical intensity per unit area is to bemaintained) or received signal intensity (if optical intensity per unitarea is permitted to fall).

The discussion of FIGS. 1a through 1e and 2a through 2d highlighted somebasic tradeoffs that may exist in a smart illumination system such asa: 1) a tradeoff between illuminated region size and received signalstrength; 2) a tradeoff between received signal strength and illuminatorpower consumption; 3) a tradeoff between illuminated region size andscanning time; 4) a tradeoff between illuminator power and the distancebetween an object of interest and the camera. An additional tradeoff mayinclude the reflectivity of the object of interest and emitted opticalpower. Here, a typical time-of-flight illuminator will emit infra-red(IR) light. If the object of interest to be illuminated substantiallyreflects IR light, the illuminator may emit less optical power. Bycontrast, if the object of interest to be illuminated does not reflectIR light particularly well, the illuminator may increase its emittedoptical power.

Which tradeoffs control and/or which direction and how heavily anyparticular tradeoff should be weighed is apt to be a function of theparticular circumstances surrounding any particular illuminationscenario.

For example, consider a situation where an object of interest to beilluminated is of modest size and is far away from the camera. Here, ifthe available power budget is large and there is a desire for a completereading in a short amount of time, a smart illumination control systemmay choose to fully illuminate the object's region with high illuminatorpower without any scanning. By contrast, in another situation in whichthe object of interest is large and is close to the camera but where theavailable power budget is small and the need for a complete readingimmediately is lacking, the same smart illumination system may choose toform a smaller illuminated region and scan it over the region ofinterest.

From these examples it should be clear that a smart illumination systemmay consider the surrounding conditions before illuminating a particularregion of interest with a specific illuminated region size, illuminatorpower and whether any scanning is to take place.

FIG. 3 shows the integration of smart illumination technology 301 into aworking computing system such as a handheld tablet or smartphone. Here,the smart illumination technology may be implemented, e.g., partially orwholly in the device driver software and/or firmware for an integratedcamera device that includes time-of-flight measurement capability. Thesoftware/firmware may be stored, e.g., in non volatile memory of thecomputing system (e.g., in FLASH firmware or system storage).

As observed in FIG. 3, the smart illumination technologysoftware/firmware may be realized as a system of methods that aredesigned to strike appropriate balances amongst the aforementionedtradeoffs given a set of input parameters that correspond to thesurrounding conditions of a depth profiling image capture sequence.

As observed in FIG. 3, the smart illumination methods 301 may accept oneor more of the following input parameters from the host system 302: 1)the object of interest (which may be specified as what the object is(e.g., hand, face, etc.) and/or characteristics of the object's locationand/or shape within the field of view); 2) how time critical thetime-of-flight measurement is (how quickly it needs to be performed);and, 3) the power budget of the time-of-flight system and/or itsilluminator (e.g., specified as a maximum permitted power). Thecomponents of the host system 302 that generate these input parametersmay include an intelligent object recognition software applicationand/or hardware logic circuit component (e.g., for facial recognition,hand recognition etc.). Power management software, firmware and/orhardware of the host system 302 may generate the power budget inputinformation.

The smart illumination methods 301 may also accept input informationfrom the camera system 303 b itself such as: 1) the distance between theobject of interest and the camera; 2) the reflectivity of the object ofinterest; 3) the location and/or shape of the object of interest; 4) theintensity of the background light. Any of the input parameters providedby the camera may be provided after initial illumination of the object(or the field of view generally). That is, e.g., as an initial responseto input from the host system 302, the time-of-flight system mayinitially illuminate the object and/or field of view as a first pass.Data collected from the first pass is then presented to the smartillumination methods 301 so that they can better optimize the capture ofthe object in terms of which regions are illuminated and how intense theemitted light should be.

With the applicable input parameters the smart illumination methods 301effectively determine which tradeoffs control and/or which direction andhow heavily any particular tradeoff should be weighed in order togenerate image capture control commands for the camera 303 b thatspecify what region(s) to illuminate, the intensity of the emittedlight, whether any scanning applies and, e.g., if so applicable scanningparameters (e.g., time to scan, velocity of scan, scanning pattern,etc.).

FIGS. 4a through 4c show another embodiment of a method that the smartillumination methods 301 of FIG. 3 may be designed to perform. Asobserved in FIG. 4a , initially, a large area 410 that, e.g.,substantially covers the camera's field of view 401 is first illuminatedby the time-of-flight illuminator. In some embodiments, as observed inFIG. 4a , the large area 410 may correspond to the entire field of view401. In other embodiments the large area 410 may correspond to asubstantial portion but less than the entire field of view 401 (e.g.,approximately 33%, 50%, 66%, 75% of the field of view 410). Here, notethat the illumination of a large area 410 within the field of view 401may correspond to weaker received optical intensities because theemitted illumination is “spread out” over a wider surface area.

The image sensor that receives the reflected light includes circuitry(such as sense amplifier circuitry) that measures the received signalintensity against some threshold at each pixel. Those of the pixels thatreceived light at a weak optical intensity are identified (e.g., thepixels whose received optical intensity fell below the threshold areidentified). In many situations, as observed in FIG. 4b , it is expectedthat groups of neighboring pixels will fall below the threshold which,in turn, corresponds to the identification of regions 411 within thefield of view 401 that received a weak optical signal.

Here, a smart illumination method 301 of FIG. 3 may receive signalintensity information for all of the image sensor's pixels and apply thethreshold to determine the size and location of regions 411, or,alternatively, may only receive the identity of the pixels that fellbelow the threshold and from them determine regions 411. Uponrecognition of the weak signal regions 411, the smart illuminationmethod will proceed, as observed in FIG. 4c , to direct commands to thetime-of-flight illuminator to re-illuminate these same regions 411.

The re-illumination is performed with more concentrated light to “boost”the optical intensity directed to the regions 411. The concentration iseffected through the formation of smaller regions of illuminated light(as compared to illuminating the entire field of view), e.g., with thesame amount of illuminator intensity that was emitted when the field ofview was flooded. With the re-illumination of these regions withstronger light, the time-of-flight measurement should be complete inthat the pixels that previously received a weak optical signal will nowreceive a sufficiently strong optical signal.

In the case of an illuminator having a movable optical component, theportions of the field of view requiring re-illumination can bere-illuminated by moving one or more optical components to direct a beamof light onto each region. In the case of an illuminator having apartitioned field of view, the portions of the field of view requiringre-illumination are re-illuminated by illuminating their correspondingpartitions. In one embodiment, the same total amount of optical powerused to initially illuminate the entire field of view may be the same asthe total amount of power used to illuminate only a partition that isre-illuminated.

FIGS. 5 through 8 a,b provide different embodiments of illuminators thatare capable of performing the smart illumination techniques discussedabove.

FIG. 5 shows an embodiment of an illuminator having a movable lensassembly 501, 502 for adjusting the size of the illuminated region (bymoving the lens vertically above the light source) as well as scanningthe illumination region or at least directing the illumination to anyarea with the camera's field of view (by tiling the plane of the lensabove the light source).

As observed in FIG. 5, a light source 503 sits beneath the lens 501 and,when illuminated, the emitted light propagates through the lens and intothe camera's field of view. The light source 503 may be implemented, forexample, as a semiconductor chip having an array of infra-red (IR)VCSELs or LEDs. The use of the array helps “boost” maximum opticaloutput power which runs approximately coextensively with NL where N isthe number of VCSELs/LEDs in the array and L is the maximum output powerof each VCSEL/LED. In an embodiment, all VCSELs/LEDs in the arrayreceive a same drive current so that the emitted optical power of eachVCSEL/LED is approximately the same as the other VCSELs/LEDs in thearray. Optical output power is controlled by controlling the magnitudeof the drive current.

A pair of voice coil motors 531, 532 each with spring return 533, 534are used as actuators to define the vertical position of each of twopoints along the outer edge of the lens 501. The tilt angle of the lens501 about the y axis is substantially defined by the force of a firstmotor 531 as applied against its return spring 533. The tilt angle ofthe lens 502 about the x axis is substantially defined by the force of afirst motor 532 as applied against its return spring 534. From thesebasic scenarios, any tilt angle for the lens can be established as afunction of the respective forces applied by the motors and thecounteractive forces applied by the springs. On opposite sides of thelens holder 501 from the return springs may exist hinged pins or balljoints, for example, to permit the lens holder 501 to pivot around boththe x and y axis.

Additionally, the vertical positioning of the lens 501 can beestablished by actuating the two motors 531, 532 equally. That is, ifboth motors 531, 532 extend an equal amount outward the lens will beraised in the +z direction. Correspondingly, if both motors 531, 532recess an equal amount inward the lens will be lowered in the −zdirection. Instead of the aforementioned hinged pins or ball joints, oneor more additional voice coil motor actuators may be located along theperiphery of the lens holder 502 to further stabilize both tilt angleand vertical positioning of the lens (e.g., three actuators spaced 120°apart, four actuators spaced 90° apart, etc.).

FIG. 6 shows an illuminator having a movable light source 603. The lightsource itself may be implemented as the light source 503 discussed abovewith respect to FIG. 5. In the approach of FIG. 6, the lens assemblypositioning remains substantially fixed but a platform or substrate 610that the light source 603 is mounted on is movable according to the sameprinciples discussed above with respect to the lens holder 502 of FIG.5. That is, a set of voice coil motor actuator and return spring pairs631/633, 632/634 can be used to effect a tilt angle of the platform 610about either or both of the x and y axes. Changing the tilt angle of theplatform changes the angle of incidence of the emitted light into thelens, which, in turn, will change the pointing direction of the beam oflight that is emitted from the lens into the camera's field of view.

A third voice coil actuator and return spring pair (not shown) may becoupled on an edge of the platform 610 other than the two edges wherevoice coil motor actuator and return spring pairs 631/633, 632/634 arelocated to effect vertical movement of the platform 610 along the zaxis, which, in turn, will affect the size of the illuminated region(spot size) in the camera's field of view.

FIG. 7 shows another illuminator embodiment in which a light source 712is affixed to the underside of a mechanical arm 713 oriented at an anglethat positions the light source to direct light to a mirror 714 mountedon a movable platform 710. A lens and lens holder are fixed in positionabove the mirror so that light that is reflected off the surface of themirror propagates through the lens and into the camera's field of view.The light source may be implemented as described above with respect toFIGS. 5 and 6.

A set of voice coil motor actuator and return spring pairs 731/733,732/734 can be used to effect a tilt angle of the platform 710 abouteither or both of the x and y axes. Changing the tilt angle of theplatform 710 changes the angle of incidence of the emitted light intothe lens, which, in turn, will change the pointing direction of the beamof light that is emitted from the lens into the camera's field of view.

A third voice coil actuator and return spring pair (not shown) may becoupled on an edge of the platform 710 other than the two edges wherevoice coil motor actuator and return spring pairs 731/733, 732/734 arelocated to effect vertical movement of the platform 710 along the zaxis, which, in turn, will affect the size of the illuminated region(spot size) in the camera's field of view.

Either of the illuminator designs of FIGS. 6 and 7 may be enhanced toinclude a movable lens arrangement as discussed in FIG. 5. Addingmovable lens capability to the designs of FIGS. 6 and 7 may, forexample, provide faster scanning times and/or larger emission anglesfrom the illuminator. Each of the movable platforms 610, 710 of FIGS. 6and 7 may be implemented as a micro-electrical-mechanical (MEM) deviceto place the light source (FIG. 6) or mirror (FIG. 7) in any location onthe xy plane.

FIGS. 8a and 8b show an embodiment of an illuminator that is designed toindividually illuminate different partitions within the field of view.As observed in FIGS. 8a and 8b the illuminator 801 includes asemiconductor chip 804 having a light source array 806_1 through 806_9for each partition within the field of view. Although the particularembodiment of FIGS. 8a and 8b show nine field of view sections arrangedin an orthogonal grid, other numbers and/or arrangements of partitionsmay be utilized. Likewise, although each light source array is depictedas a same sized N×N square array other array patterns and/or shapesincluding different sized and/or shaped arrays on a same semiconductordie may be utilized. Each light source array 106_1 through 106_9 may beimplemented, for example, as an array of VCSELs or LEDs.

Referring to FIGS. 8a and 8b , in an embodiment, the illuminator 801also includes an optical element 807 having a micro-lens array 808 on abottom surface that faces the semiconductor chip 804 and having anemission surface with distinct lens structures 805 for each partition todirect light received from its specific light source array to itscorresponding field of view partition. Each lens of the micro-lens array808 essentially behaves as a smaller objective lens that collectsdivergent light from the underlying light sources and shapes the lightto be less divergent internal to the optical element as the lightapproaches the emission surface. In one embodiment, there is amicro-lens allocated to and aligned with each light source in theunderlying light source array although other embodiments may exist wherethere is more or less micro-lenses per light source for any particulararray.

The micro-lens array 808 enhances optical efficiency by capturing mostof the emitted optical light from the underlying laser array and forminga more concentrated beam. Here, the individual light sources of thevarious arrays typically have a wide emitted light divergence angle. Themicro-lens array 808 is able to collect most/all of the diverging lightfrom the light sources of an array and help form an emitted beam oflight having a smaller divergence angle.

Collecting most/all of the light from the light source array and forminga beam of lower divergence angle essentially forms a higher opticalbower beam (that is, optical intensity per unit of surface area isincreased) resulting in a stronger received signal at the sensor for theregion of interest that is illuminated by the beam. According to onecalculation, if the divergence angle from the light source array is 60°,reducing the emitted beam's divergence angle to 30° will increase thesignal strength at the sensor by a factor of 4.6. Reducing the emittedbeam's divergence angle to 20° will increase the signal strength at thesensor by a factor of 10.7.

The optical element 807 may additionally be designed to provide furtherdiffusion of the collected light by, e.g., constructing the element 807with materials that are translucent in the IR spectrum and/or otherwisedesigning the optical path within the element 807 to impose scatteringinternal reflections (such as constructing the element 807 as amulti-layered structure). As mentioned briefly above, the emissionsurface of the optical element 807 may include distinctive lensstructures 805 each shaped to direct light to its correct field of viewpartition. As observed in the specific embodiment of FIGS. 8a and 8b ,each lens structure 805 has a rounded convex shape. Other embodimentsmay have, e.g., sharper edged trapezoidal shapes or no structure at all.

The optical element 807 may also be movable, e.g., by mechanicallycoupling, e.g., two or three voice-coil motor actuator and return springpairs consistent with the discussions provided above with respect toFIGS. 5, 6 and 7. By designing the optical element 807 to be movable,scanning a single beam within a partition as discussed above withrespect to FIG. 2d can be achieved by moving the optical element 807 ina scanning motion while illuminating only the light source associatedwith the partition being scanned. Additionally, the simultaneousscanning of multiple partitions as observed in FIG. 2e can beaccomplished by illuminating each of the partitions' respective lightsources and moving the optical element 807 in a scanning motion.

FIG. 9 shows an integrated traditional camera and time-of-flight imagingsystem 900. The system 900 has a connector 901 for making electricalcontact, e.g., with a larger system/mother board, such as thesystem/mother board of a laptop computer, tablet computer or smartphone.Depending on layout and implementation, the connector 901 may connect toa flex cable that, e.g., makes actual connection to the system/motherboard, or, the connector 901 may make contact to the system/mother boarddirectly.

The connector 901 is affixed to a planar board 902 that may beimplemented as a multi-layered structure of alternating conductive andinsulating layers where the conductive layers are patterned to formelectronic traces that support the internal electrical connections ofthe system 900. Through the connector 901 commands are received from thelarger host system such as configuration commands that write/readconfiguration information to/from configuration registers within thecamera system 900. Additionally, the commands may be any commandsassociated with a smart illumination technology system such as any ofthe outputs provided by the smart technology methods 301 discussed abovewith respect to FIG. 3

An RGBZ image sensor 903 is mounted to the planar board 902 beneath areceiving lens 904. The RGBZ image sensor 903 includes a pixel arrayhaving an RGBZ unit pixel cell. The RGB pixel cells are used to supporttraditional “2D” visible image capture (traditional picture taking)functions. The Z pixel cells are sensitive to IR light and are used tosupport 3D depth profile imaging using time-of-flight techniques.Although a basic embodiment includes RGB pixels for the visible imagecapture, other embodiments may use different colored pixel schemes(e.g., Cyan, Magenta and Yellow).

The image sensor 903 may also include ADC circuitry for digitizing thesignals from the image sensor and timing and control circuitry forgenerating clocking and control signals for the pixel array and the ADCcircuitry.

The planar board 902 may include signal traces to carry digitalinformation provided by the ADC circuitry to the connector 901 forprocessing by a higher end component of the host computing system, suchas an image signal processing pipeline (e.g., that is integrated on anapplications processor).

A camera lens module 904 is integrated above the RGBZ image sensor 903.The camera lens module 904 contains a system of one or more lenses tofocus received light to the image sensor 903. As the camera lensmodule's reception of visible light may interfere with the reception ofIR light by the image sensor's time-of-flight pixel cells, and,contra-wise, as the camera module's reception of IR light may interferewith the reception of visible light by the image sensor's RGB pixelcells, either or both of the image sensor's pixel array and lens module903 may contain a system of filters arranged to substantially block IRlight that is to be received by RGB pixel cells, and, substantiallyblock visible light that is to be received by time-of-flight pixelcells.

An illuminator 905 capable of illuminating specific regions within thefield of view consistent with the smart illumination technology asexplained in the above discussions is mounted on the planar board 902.The illuminator 905 may be implemented, for example, as any of theilluminators discussed above with respect to FIGS. 5 through 8 a,b. Alight source driver is coupled to the illuminator's light source 907 tocause it to emit light with a particular intensity and modulatedwaveform.

In an embodiment, the integrated system 900 of FIG. 9 supports threemodes of operation: 1) 2D mode; 3) 3D mode; and, 3) 2D/3D mode. In thecase of 2D mode, the system behaves as a traditional camera. As such,illuminator 905 is disabled and the image sensor is used to receivevisible images through its RGB pixel cells. In the case of 3D mode, thesystem is capturing time-of-flight depth information of an object in thefield of view of the illuminator 905. As such, the illuminator 905 isenabled and emitting IR light (e.g., in an on-off-on-off . . . sequence)onto the object. The IR light is reflected from the object, receivedthrough the camera lens module 1504 and sensed by the image sensor's Zpixels. In the case of 2D/3D mode, both the 2D and 3D modes describedabove are concurrently active.

FIG. 10 shows a depiction of an exemplary computing system 1000 such asa personal computing system (e.g., desktop or laptop) or a mobile orhandheld computing system such as a tablet device or smartphone. Asobserved in FIG. 10, the basic computing system may include a centralprocessing unit 1001 (which may include, e.g., a plurality of generalpurpose processing cores) and a main memory controller 1017 disposed onan applications processor or multi-core processor 1050, system memory1002, a display 1003 (e.g., touchscreen, flat-panel), a local wiredpoint-to-point link (e.g., USB) interface 1004, various network I/Ofunctions 1005 (such as an Ethernet interface and/or cellular modemsubsystem), a wireless local area network (e.g., WiFi) interface 1006, awireless point-to-point link (e.g., Bluetooth) interface 1007 and aGlobal Positioning System interface 1008, various sensors 1009_1 through1009_N, one or more cameras 1010, a battery 1011, a power managementcontrol unit 1012, a speaker and microphone 1013 and an audiocoder/decoder 1014.

An applications processor or multi-core processor 1050 may include oneor more general purpose processing cores 1015 within its CPU 1001, oneor more graphical processing units 1016, a main memory controller 1017,an I/O control function 1018 and one or more image signal processorpipelines 1019. The general purpose processing cores 1015 typicallyexecute the operating system and application software of the computingsystem. The graphics processing units 1016 typically execute graphicsintensive functions to, e.g., generate graphics information that ispresented on the display 1003. The memory control function 1017interfaces with the system memory 1002. The image signal processingpipelines 1019 receive image information from the camera and process theraw image information for downstream uses. The power management controlunit 1012 generally controls the power consumption of the system 1000.

Each of the touchscreen display 1003, the communication interfaces1004-1007, the GPS interface 1008, the sensors 1009, the camera 1010,and the speaker/microphone codec 1013, 1014 all can be viewed as variousforms of I/O (input and/or output) relative to the overall computingsystem including, where appropriate, an integrated peripheral device aswell (e.g., the one or more cameras 1010). Depending on implementation,various ones of these I/O components may be integrated on theapplications processor/multi-core processor 1050 or may be located offthe die or outside the package of the applications processor/multi-coreprocessor 1050.

In an embodiment one or more cameras 1010 includes an illuminatorcapable of illuminating specific regions within the camera field of viewconsistent with the smart illumination technology as explained in theabove discussions. Application software, operating system software,device driver software and/or firmware executing on a general purposeCPU core (or other functional block having an instruction executionpipeline to execute program code) of an applications processor or otherprocessor may direct smart illumination commands, or other commands, toand receive image data from the camera system. Other commands that maybe received by the camera 1010 include commands for entrance into orexit from any of the 2D, 3D or 2D/3D camera system states discussedabove.

The smart illumination technology itself may be implemented, partiallyor wholly, as any one or more of the following: 1) software that runs ona general purpose processing core; 2) system firmware (e.g., BIOSfirmware); 3) dedicated logic circuitry (e.g., disposed at one or moreof the following: on the camera 1010, integrated in an ISP 1019;integrated with an I/O or peripheral controller 1018). As discussedabove the smart illumination technology may receive input informationfrom the power management control unit which itself may be implemented,partially or wholly, with any one or more of software that runs on ageneral purpose processing core, system firmware, dedicated logiccircuitry, etc.

Embodiments of the invention may include various processes as set forthabove. The processes may be embodied in machine-executable instructions.The instructions can be used to cause a general-purpose orspecial-purpose processor to perform certain processes. Alternatively,these processes may be performed by specific hardware components thatcontain hardwired logic for performing the processes, or by anycombination of programmed computer components and custom hardwarecomponents.

Elements of the present invention may also be provided as amachine-readable medium for storing the machine-executable instructions.The machine-readable medium may include, but is not limited to, floppydiskettes, optical disks, CD-ROMs, and magneto-optical disks, FLASHmemory, ROMs, RAMs, EPROMs, EEPROMs, magnetic or optical cards,propagation media or other type of media/machine-readable mediumsuitable for storing electronic instructions. For example, the presentinvention may be downloaded as a computer program which may betransferred from a remote computer (e.g., a server) to a requestingcomputer (e.g., a client) by way of data signals embodied in a carrierwave or other propagation medium via a communication link (e.g., a modemor network connection).

In the foregoing specification, the invention has been described withreference to specific exemplary embodiments thereof. It will, however,be evident that various modifications and changes may be made theretowithout departing from the broader spirit and scope of the invention asset forth in the appended claims. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense.

The invention claimed is:
 1. A method, comprising: illuminating a regionwithin a time-of-flight camera system's field of view; determining depthprofile information within said field of view using time-of-flightmeasurement techniques comprising sensing a reflection from theilluminating with a plurality of pixels and measuring a respective timebetween the illuminating and the sensing for each of the plurality ofpixels; determining one or more regions within said field of view wherea received signal from said illuminating was weak; re-illuminating saidone or more regions with stronger light than said one or more regionsreceived during said illuminating, each of said one or more regionsbeing smaller than said region; and, determining depth profileinformation within said one or more regions using time-of-flightmeasurement techniques comprising sensing a reflection from there-illuminating with a plurality of pixels within the one or moreregions and measuring a respective time between the re-illuminating andthe sensing of the reflection from the re-illuminating for each pixelwithin the one or more regions.
 2. The method of claim 1 wherein saidregion is said time-of-flight's entire field of view.
 3. The method ofclaim 1 wherein said re-illuminating further comprises re-illuminating apartition with said field of view.
 4. The method of claim 3 wherein alight source that illuminates said partition emits stronger light duringsaid re-illuminating than during said illuminating.
 5. The method ofclaim 1 wherein said re-illuminating further comprises directing lightdirectly to each of said one or more regions.
 6. The method of claim 5wherein said light is shaped in size to respectively encompass each ofsaid one or more regions during said re-illuminating.
 7. The method ofclaim 6 wherein a light source that illuminates said region and said oneor more regions does not emit stronger light when re-illuminating atleast one of said one or more regions than when illuminating saidregion.
 8. A non transitory machine readable storage medium comprisingstored program code that when processed by a digital processing systemcauses a method to be performed by a time-of-flight camera system, saidmethod comprising: illuminating a region within a time-of-flight camerasystem's field of view; determining depth profile information withinsaid field of view using time-of-flight measurement techniquescomprising sensing a reflection from the illuminating with a pluralityof pixels and measuring a respective time between the illuminating andthe sensing for each of the plurality of pixels; determining one or moreregions within said field of view where a received signal from saidilluminating was weak; re-illuminating said one or more regions withstronger light than said one or more regions received during saidilluminating, each of said one or more regions being smaller than saidregion; and, determining depth profile information within said one ormore regions using time-of-flight measurement techniques comprisingsensing a reflection from the re-illuminating with a plurality of pixelswithin the one or more regions and measuring a respective time betweenthe re-illuminating and the sensing of the reflection from there-illuminating for each pixel within the one or more regions.
 9. Themachine readable storage medium of claim 8 wherein said region is saidtime-of-flight's entire field of view.
 10. The machine readable storagemedium of claim 8 wherein said re-illuminating further comprisesre-illuminating a partition with said field of view.
 11. The machinereadable storage medium of claim 10 wherein a light source thatilluminates said partition emits stronger light during saidre-illuminating than during said illuminating.
 12. The machine readablestorage medium of claim 8 wherein said re-illuminating further comprisesdirecting light directly to each of said one or more regions.
 13. Themachine readable storage medium of claim 12 wherein said light is shapedin size to respectively encompass each of said one or more regionsduring said re-illuminating.
 14. The machine readable storage medium ofclaim 13 wherein a light source that illuminates said region and saidone or more regions does not emit stronger light when re-illuminating atleast one of said one or more regions than when illuminating saidregion.
 15. An apparatus, comprising: a time-of-flight camera systemcomprising: an illuminator to illuminate a region within saidtime-of-flight camera system's field of view; an image sensor comprisingpixels to receive optical signals from said illumination for determiningdepth profile information within said field of view through measurement,for each of the pixels, of a respective time between illumination by theilluminator and the reception of the optical signals, said image sensorhaving circuitry to determine one or more regions within said field ofview where a received optical signal from said illuminating was weak;said illuminator to also re-illuminate said one or more regions withstronger light than said one or more regions received during saidilluminating, each of said one or more regions being smaller than saidregion; and, said image sensor to also receive optical signals from saidre-illumination for determining depth profile information within saidone or more regions through measurement, for a set of the pixels withinthe one or more regions, of a respective time between re-illumination bythe illuminator and the reception of the optical signals from there-illumination.
 16. The apparatus of claim 15 wherein said one-or-moreregions are defined with pixels of said image sensor.
 17. The apparatusof claim 15 wherein said region is said time-of-flight's entire field ofview.
 18. The apparatus of claim 15 wherein said illuminator has amovable optical component to individually direct said re-illumination tosaid one or more regions.
 19. The apparatus of claim 15 wherein saidilluminator has an optical component and a light source designed topartition illumination of said time-of-flight camera's field of view.20. The apparatus of claim 18 wherein said re-illumination of said oneor more regions is performed by individually illuminating respective oneor more partitions where said one or more regions reside.
 21. Acomputing system, comprising: an applications processor having aplurality of processing cores and a memory controller, a system memorybeing coupled to said memory controller; a time-of-flight camera systemcoupled to said applications processor, said time-of-flight camerasystem including: an illuminator to illuminate a region within saidtime-of-flight camera system's field of view; an image sensor to receiveoptical signals from said illumination for determining depth profileinformation within said field of view using time-of-flight measurementtechniques, said image sensor having circuitry to determine one or moreregions within said field of view where a received optical signal fromsaid illuminating was weak through measurement, for each of the pixels,of a respective time between illumination by the illuminator and thereception of the optical signals, said image sensor having circuitry todetermine one or more regions within said field of view where a receivedoptical signal from said illuminating was weak; said illuminator to alsore-illuminate said one or more regions with stronger light than said oneor more regions received during said illuminating, each of said one ormore regions being smaller than said region; and, said image sensor toalso receive optical signals from said re-illumination for determiningdepth profile information within said one or more regions throughmeasurement, for a set of the pixels within the one or more regions, ofa respective time between re-illumination by the illuminator and thereception of the optical signals from the re-illumination.
 22. Thecomputing system of claim 21 wherein said one-or-more regions aredefined with pixels of said image sensor.
 23. The computing system ofclaim 21 wherein said region is said time-of-flight's entire field ofview.
 24. The computing system of claim 21 wherein said illuminator hasan optical component and a light source designed to partitionillumination of said time-of-flight camera's field of view.
 25. Thecomputing system of claim 21 wherein said re-illumination of said one ormore regions is performed by individually illuminating respective one ormore partitions where said one or more regions reside.
 26. The computingsystem of claim 21 wherein said illuminator has a movable opticalcomponent to individually direct said re-illumination to said one ormore regions.